fix: detect a worker that exits while steps are still assigned to it - #1094

Closes #1087.

The hang

A worker taking the data-drop path runs _handle_stop_command, breaks its loop, and exits with code 0. find_dead_workers filters on exitcode not in {None, 0}, so that exit is invisible — the step dispatched to the worker stays in currently_running_steps, no result ever arrives, and the run loop waits on a process that is gone. As the issue notes, the new stall guard treats an in-flight step as progress on purpose, so this hang class needs its own detection.

The change

WorkerManager gains assigned_steps: dict[UUID, set[UUID]], recorded by multi_execute_step immediately after send_command — which is also the path that reuses existing queues with no liveness check, so a step dispatched to an already-dead process is covered by the same ledger.

find_orphaned_steps() returns (cfw_uuid, exitcode, still-owed step uuids) for every worker whose process has exited, and _check_for_error raises:

MlodaRunError: Worker process(es) exited with steps still assigned:
cfw <uuid> exited with code 0 owing step(s) <uuid>, <uuid>

Two deliberate choices:

• Any exitcode counts, including 0. Once a process has exited it will never produce a result, so an assigned step with no result is lost whatever the code. Restricting to 0 would just move the blind spot.
• Checked against result_uuids_collection. A worker that finished its work and then exited owes nothing and is not reported, so a normal shutdown stays quiet.

Verification

8 tests in TestWorkerManagerOrphanedStepDetection covering: clean exit owing a step, abnormal exit owing a step, result-already-arrived, partially-drained worker, live worker, exited worker owing nothing, assignment accumulation across dispatches, and multiple affected cfws.

Mutation check — restored the original predicate (if exitcode is None or exitcode == 0: continue), i.e. reintroduced exactly the bug:

FAILED ...::test_a_clean_exit_owing_a_step_is_reported
FAILED ...::test_only_the_steps_still_owed_are_named
FAILED ...::test_record_assignment_accumulates_across_dispatches
FAILED ...::test_each_exited_worker_is_reported_separately
4 failed, 40 passed

The clean-exit test also asserts find_dead_workers() == [] inline, so if that check ever starts catching this case the test says so instead of quietly duplicating coverage.

tests/test_core: 5201 passed, 4 xfailed, run serially.

Two notes

A test-only fix rode along. TestMultiExecuteStep builds Mock(spec=FeatureGroupStep) / Mock(spec=TransformFrameworkStep) steps that never stubbed get_uuids(). multi_execute_step now reads it, and a bare Mock is not iterable, so 5 tests there failed — for that reason and no other. Each now returns real uuids, matching what a real step does.

-n 8 is unreliable on my machine. xdist intermittently dies with an INTERNALERROR ... assert not crashitem on a different test each run (test_deep_chain_ancestors_no_recursion_error, then test_class_source_hash_getsource_wrong_text_identical_bodies_hash_equal), which cascades into ~30 spurious failures. Serial runs are clean, so the numbers above are serial. Flagging in case you see the same locally — it looks like a worker-crash flake rather than anything in this change.

ruff format --check --line-length 120, ruff check, mypy --strict --ignore-missing-imports all clean.